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fy Kombiniertes Rontgendurchleuchtungs- und Laserprojektionsgerat 

Mobile r C-Bogen Rorrtgenappa ret mlt integrierten La- 
serprojektoren zur Durchleuchtung und oberflechlichen 
Darstellung der gerontgten Strukturen mittols Laser- 
strahltechnik. : 
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Kombiniertes RSntg endurchleuchtungs- und Laserproiektionsgerat 

Die Erfindung bezieht sich auf ein kombiniertes RontgendurcUeuchtungs- und 
Laseiprojektionsgerat zur genauen Lokalisiening und oberflachlichen Darstellung 
gerontgter Strukturen. Dieses bewirkt eine Zdtersparnis und vermeidet unnStige 
Strahlungsbelastung des Patienten und medizinischen Personals. 

■ ' 

Stand der Technik im Bereich der heutigen Onrurgie ist es, zunehmend minimal 
mvasiv vorzugehen, d.h. Jdein dimensionierte Zugange zu wahlen, urn den 
geplanten operativen Eingriff (toclizufubren. Mit Hilfe modemster 
Computertechnik wird eine bis dato unbekannte Genauigkeit bei der 
praoperativen Planung und der Durchfuhrung erreicht. Diese neuen Methoden 
werden dem Oberbegriff der Computer Assistierten Onrurgie (engl. Computer 
Assisted Surgery = CAS) zugeordnet. Beispiele bierfur sind zB. der Robodoc 
welcher die millimeter genaue Frasung fur eine Huftschaftprothese ermogticht, 
oder Naviganonssysteme, um z.B. Schrauben hn Wirbelsaulenbereich gefahrlos 
einbringen zu kdnnen, ohne das Ruckenmark zu schadigen. In beiden Beispielen 
ist meistens eine ausfuhrliche Besumdsaumahme der Patientendaten nrittels CT 
oder MRT neben einer praoperativen Planung notwendig! Spezielle Gerate sind 
im OP zur Dmchfuhrung airizustellen. Fur einige Anwendungen stent der 
zeimche, apparajiye und rechnerische Aufwand jedoch nicht in optimalem 
Verhalmis zum gewonnenen Nutzen. - 

* - - t * 

- ... t ^ . * • i _ . ^ • 

. * *"**. , *« "« ***' **' * * 

• • • • • - • •*•««.. 

. - ■ . . - . - * . . . ■ . ■ 
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Gerataufbau: 

* • V ."**••• - '- . '■ • »..•-. J ■ . ' • 

- "■• • ■ • * '. ' " • . : • ". * ' - * -■ •.**• • •- . ■ • 

■ ■ Z * * ** ' ■* • • r * „■*' / •""**** 

- . ■ '•.•'.'!■.." . " i . ■'. ■ . " . . ■ 

... • • .' • * • • ■ • . * 

Die beschriebene Erfindung kann pperatiye Eingriffe bei erhohter Genauigkeit 
verkurzen und gleicbzeitig die intraoperative StraUenbelastune senken. 
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Grundsatzlich ist das in Fig. 1 dargestellte kombinierte Rontgendurchleuchtungs- 
und Laserprojektionsger&t (C) folgendermaBen aufgebaut: 
Auf def Rflckseite des Bildverstarkers (BV) und der Rontgenrahre (RR) sind 
jeweils eine Lasereinheit (LI und L2) mit integriertem 
Ultraschallentferaungsmesser (Ul und U2) mqntiert Um die Laserprojektiori zu 
nutzen, muB die entsprechende Lasereinheit zur projezierenden Oberflache, zB. 
der Patientenhantoberflacbe, zeigen. Dieses geschieht enlweder dadurct, dafi BV 
und RR uber Gelenke (Gl und G2) um 180' drehbar montiert sind oder dadurch, 
dafi uber eine Mechanik die Lasereinheit vor BY oder RR positionieit wird, 
ahnlich einer automatischen Dachoffrmng bei einem Cabriolet-PKW. Auch ist es 
moglich, die Laser-/Ultraschalleinheit permanent neben BV oder RR 

* * . . • * ■ 

anzubringen/ C-Bogen Rontgendurchleuchtungsgerat, Laser und 
UltraschaUentfenungsmesser sind gemeinsam uber einen Computer verbundeiL 

Dieser dient der Benutzereingabe und der ultraschalluntersttttzten 

. ■ ■ . ■ . 

■ . - • . 

Laserprojektionssteuerung. 

■ . 

■ " ■" ■ ...... 

Beispielanwendnngen: 

• • . * ' - ■ ■ ■ ' '-' ' ' ' = 

• . - • - • • ■ • ' '. • - 

• • . . ... - ■ . . • ' .' • 

1 . Minimalinvasive Plattenosteosynthese 

In einem ersten Durchleuchtungsbild wird der zu versorgende 
Frakturbereich auf dem Computermonitor darjgestellt Die Fraktur und die 
Knochen kdnnen im Computer markiert und mittels I^erstraMtechnik auf 

die Korperoberfkache projeziert werden. Die Konturen kdnnen mil einem 

■ • "... • . * ... ■ ■ ■ . - j .. ■ 

Filzstift nachgezeichnet werden. Reoposition- Das passende 
Plattenimplantat wird nun im Conrputer im Sinne einer intraoperatiyen 
Planung ausgewahlt, in das bestehende Bild eingeladen und an die auf dem 
Monitor dargestellten Knochenfragmente angelegt bis eine 
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zufiiedenstelleride Posititon erreicht ist Die Platte wird mit Stichinzisioii 

fiber dern Knochen positkmiert Emeute Durchleuchtung. Die Platte iait 

■ ■ - ... • ■ 

den Lochern wird auf die Korperoberflache projeziert, Stichinzisionen 
werden genau iiber den Lochern gemacht, Fixierung mit Schrauben. 



2. Marknagelveniegelung /-entriegelung 

Minimal invasiv werden nach Lokalisierung und Laseiprojektion auf die 
Patientenhaut Stichinzisionen mit Venieghmgsbolzen eingefiihrt/ entfemt 



3. Schraubenosteosynihese / Materialentfernung (ME) 

Lokalisierung der Implantate, Las^projektion, Stidiinzisionen. 

■ - 

anschliefiend Osteosynthese/ME. 



4. K-Drahtosteosynthesen 

Die Verlaufsrichtung der K-Drahte kann angezeigt werden, ran ein genaues 
„SchieBen" der K-Drahte zu ennoglichen. 



5. Hilfslinien anzeigen als Orientierungs- und Lokalisierungshilfe 

bei Osteotomien, Fixateur externe Anlage, WirbelsMenchirurgie, Tumor- 

. • •■ . ■ . ■ ■-, • . . • . . . . • . • . • 

. - . * - . . .... . • 

und Fremdkarperlpkalisiening, NeuroVGef^chinirgie (Lokalisation von 
Gefefianomalien nach Kontrastmittelgabe), HNO Qperationen, 
interventionelle Radiologie (Punktionen, PE, etc.), Urologie, Gynakologie 
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Anwendung 

1 +."*** * 

. • ! 

■ » ■ i ' ■ • . - » 

C-Bogen, Lasereinheiten und Entfernungsmesser sind mit einem Computer 
verbunden. Eine spezielle Software wird aufgerufen. Nach DurcMeuchtung mit 
dem C-Bogen (Fig. 2) erscheint das computertechnisch geometrisch entzerrte 

* • , ■ * * 

Rontgenbild auf dem Computemionitor. Die zuprojezierenden Strukturen werden 
maiidert oder in das Bild als voibestehende Schablone eingeladeiL Eine 
intraoperative Planting laBt sich somit durchfuhren. Es konnen im Computer auch 
> Langen und Winkel gemessen und dargestellt werden; besonders wichtig zJB. fur 
Umstellungsosteotomien, Hilfslioieiiprojektion o.a. 

Nachdem die Aufforderung zur Laserprojektion durch den Benutzer erfolgt, wird 
der Ultraschallentfernungsmesser aktiviert Die genaue Entfernung zur 

KOrperoberfl&che wird fur eine mafistabsgetreue Abbildung gemessen, 

• . • . ■- * • . ■ ... 

AnschlieBend wird die Lasereinheit angesteuert, und die Laserprojektion erfolgt 

* , - ■"■.'*" - , ... ■ 

(Fig.3). Bei Bedarf kann eine gleichzeitige Projektion beider Lasereinheiten 
erfolgen. ■■■■} +.■.■■-'■:'?■■> , 

■ ..... •_ ' ■ . . . . 
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Ansprfiche 



1 Mobiler C-Bogen Rontgenapparat mit integrierten laseiprojektoren zur 




mittels Laserstrahltechnik; 



- 

2 ; Kombiniertes R&rtgendurchleuchtungs- und Laserprojektionssystem nach 
Anspruch 1, dadurch gekennzeichnet, dafl die Laseiprojektoren auf der 
Ruckseite des Bildverst&kers und der Strahlenquelle angeordnet sind. 



Kombiniertes ROntgendurchleuchtungs- und I^eirprojektionssystem nach 
Anspruch 1, dadurch gekennzeichnet, daB die Laseiprojektoren neben 




* » 



4. Kombiniertes Rfintgendurchleuchtungs- und I^erprojektionssystem nach 

• • ' ■ .' . ■ ■ ■ ■ • ■ ■ • ■ . .-".*"".•-'"'-.'.*»' 

einem der Anspruche 1-3, dadurch gekennzeichnet, daB die Lasereinheiten 
manuell oder maschrrieU vor den Bildveistarker oder RSntgenrohrenanteil 
positioniert werden, urn die Laserprojektibn zu ennfigHchen, ^ 



5. : ; Kombiniertes Rontgendurchleuchtungs- 

•' * ■ .- ..... • ' - 

einem der AnsprQche 1-2, dadurch ge 
/ Strahlenquelle um 1 80 ' beweghch montiert sind; Diese konnen xnani 

• •• • ■ ; • ■ ■ • • . - 1. . .■ * • • . •. 

oder masclunell gedreht werdeo, iim die Laseiprojektiori zu enno^lichen. 
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6. Kombiniertes RontgendurcMeuchtirags- und Laseiprojektionssystem nach 
einem der Anspruche 1-5, dadurch gekermzeichnet, daB sich in den 
Lasereinheiten jeweils ein Ultraschall (US) -Entfenmngsmesser befindet, 
urn den Abstand zwischen Laserprojektor und Projektionsflache zu 
messen. Die US-Entfemungsmesser sind mit einem Computer verbunden, 
welcher die weitere datentechnische Verarbeitung in Verbindung mit der 
Lasersteuersoftware ermoglicht 



7. Kombiniertes Rontgendurchleachtungs- und Laserprojektionssystem nach 

- 

einem der Anspruche 1-6, dadurch gekennzeichnet, daB das 
RontgendOTcMeuchtungsgerat (C-Bogen) mit einem Computer verbunden 

- * " 

ist, welcher das Durchleuchttmgsbild mittels einer Video Capture Card 

* * 

(JFramegrabber Card") zu weiteren datentechnischen Verarbeitung in 

* * . ■ ■ • * * 

• . ■ ' • * 

Verbindung mit einer Lasersteuersoftware ennoghcht 



8. Kombiniertes ROntgendurchleuchtungs- imd Laserprojektionssystem nach 

* • * ■ *■■-•.-'• < . '■•»* 

*-.■■'. - ■. ■ * . ■ ■ ■ 

einem der Anspruche 1-7, dadurch gekennzeichnet, dafl die Lasereinheiten 
mit einem Computer verbunden sind, welcher eine exakte Positionierung 
der Laserstrahlen. mit einer Steuerungssoftware erm6glicht. 
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>- ; und Laserprojektionssystem nach 

• • . - ■ ■ ■ • » f - 

einem der Anspruche 1-8, dadurch gekennzeichnet, daB auf j 

• ...... - • . ■ '.■ • 

Computer die zu projezierenden Strukturen markiert bzw. einge 
werdenkGnne^ um sie dann auf die Koiperobeiflache mprojeziereri. 
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© A display system for visualization of body structures during medical procedures. 



@)v An/ interactive display system superimposes 
radiological images on a semi-transparent 
screen through which a surgeon views a patient 

■ during a medical procedure. The superimposed 
image is derived from image data obtained with 
an imaging system. The radiological image is 

V registered with the surgeon's view of the patient 
and displayed in real-time during a medfcaJ 
procedure. This allows the surgeon to view 
internal and external structures and the relation 
between them simultaneously, and adjust the 
procedure accordingly. A second embodiment 
employs stereoscopic viewing methods to pro- 
vide three-dimensional representations of the 

...radiological images superimposed on the 

; semi-transparent screen through which the sur- 
geon views the patient. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a system for aid- 
ing a medical practitioner, such as a surgeon, in vis- 
ualizing anatomical structures during medical proce- 
dures and more specifically to a system which allows 
the practitioner to view, in real time, both internal and 
external anatomical structures. 

2. Discussion of Prior Art 

This application is related to International appli- 
cation US 94/02572 "Computer Graphic and Live Vid- 
eo System for Enhancing Visualization of Body Struc- 
tures During Surgery* and to European Patent appli- 
cations 0549183 "System For Displaying Solid Cuts 
For Surfaces of Solid Models', 0549189 "Solid Mod- 
els Generation By Span Method Using Dividing 
Cubes" and 0549182 "Apparatus and Method For 
Displaying Surgical Cuts in Three-Dimensional Mod- 
els". ' ' 

Presently, during surgery and other medical pro- 
cedures such as endoscopy, biopsy and implantation, 
physicians view several static radiographic views of 
the patient in the operating room. Typically these are 
transparent film renderings of magnetic resonance 
(MR), computed tomography (CT), conventional X- 
ray images or ultrasound images. Since these im- 
ages are two dimensional static images, the physi- 
cians must determine actual three-dimensional (3D) 

location and shape of desired internal structures 
within the patient from the 2D images which they are 
viewing. Conceptually, the surgeon constructs a 3D 
model of the internal structures and correlates these 
internal structures with visible external structures of 
the patient where they must cut This is of ten difficult * 
because the scale and the orientation of the 2D image 
may differ from what the surgeon is seeing, and the 
surgeon may not be able to view both the patient and 
the medical diagnostic images simultaneously. • 

Another technique employed in localization of in- 
ternal structures during surgery is known as stereo- 
' tactic surgery as described in "Interactive Stereotac- 
tic Surgical Systemforthe Removal of Intracranial Tu- 
mors Utilizing the COj Laser and CT-Derived Data- 
base" by B. A. Kail. P. J. Kelly. and S. J. Goerss, IEEE 
Transactions on Biomedical Engineering, vol. BME- 
32, no. 2, pp 112-116, 1985; and 'Comprehensive 
Computer-Assisted Data Collection Treatment Plan- 
ning and Interactive Surgery" by B. A. Kali, P. J Kelly, 
and S. J. Goerss, Medical Imaging, vol. 767 pp. 509- 
514, 1987. With this approach, a rigid mechanical 
frame is attached to the patient before a CT or MR 
procedure. The frame and its landmarks can be seen 
in the resulting images. Mechanisms on the frame 
position a probe at specific location within the image. 



The disadvantages of this approach are that the 
frame limits access to the patient, and the images are 
static images which do not follow the patient .f he 
moves during surgery. 
5 A third technique used for localization of internal 

structures is described in "A Frameless Stereotaxic 
Operating Microscope for Neurosurgery" by E M. 
Friets J. W. Strohbehn, J. F. Hatch, and D. W. Ro- 
berts, IEEE Transactions on Biomedical Engmeenng, 
to vol. 36., no. 6, pp 608-617, June 1989. 

Three dimensional models of anatomical struc- 
tures can be created from data of different medical 
imaging modalities as described in the applications 
listed above. These applications describe creating 
f 5 and manipulating models of internal structures of pa- 
tients and providing images of selected structures at 
desired orientations to an operator. These allow visu- 
alization of internal structures as solid models. 

Currently there is a need for a system to aid 
20 physicians in surgery and other medical procedures 
which interactively displays computer generated rep- 
resentations of internal structures in correct relation 
with external structures of the patient. 

25 SUMMARY OF THE INVENTION 

A feature of the present invention is to provide a 
system which aids in surgery by simultaneously su- 
perimposing an image of internal structures upon ex- 
30 ternal structures even if the patients changes their 
position or if the operators change their viewing an- 

Another feature of the present invention is to pro- 
vide an interactive system which displays desired m- 
35 ternal structures upon external structures, having 
the same scale and viewed from the same orientation 
angles. : 

Another features of the present invention te to 
provide a surgeon a -heads up" display of radiologic^ 

40 images superimposed upon a patient within h.s field 

of view. ■ ■ ' . i ■ .. 

A real-time surgery apparatus for displaying in-- 

. teractive internal and external images of a patient 
employs a semi-transparent display screen allowing 
45 an operator to simultaneously view exposed surfaces 
and computer generated images of a patient 

A medical imaging device obtains three- 
dimensional (3D) imaging data of internal structures 
of the patient which are fed to a workstation. The 
so workstation creates three-dimensional <3D) comput- 
er generated models which may be manipulated wrth- 
out further need for the medical imaging device. Two- 
. dimensional computer generated images of the mod- 
els are interactively oriented and scaled so that their 
55 display on a semi-transparent display device coin- 
cides with the operator's visual image of the patiemV 
The three-dimensional position and orientation of 
the patient, display device and operator's eyes are 
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monitored in real-time. This information is then used Operation Guide, Ascension Technology Corporation, 

to adapt the computer generated display on the semi- July 5, 1 992, in the Introduction pp. 1 -3, Appendix 1 , 

transparent screen so that the visual and computer p. 89 and Appendix 3, p. 93. Tracking device 50 also 

generated images consistently coincide. determines a location (in Cartesian coordinates) of 

5 operator . 350 and semi-transparent screen 250 with 

BRIEF DESCRIPTION OF THE DRAWINGS relation to patient 1 . Patient 1 is assumed to be at the 

origin of the Cartesian coordinate system (x,y,z) = 
The features of the invention believed to be novel (0,0,0), therefore all distances relative to the patient 
are set forth with particularity in the appended claims. are simply the (x,y,z) location. The location and ori- 
The invention itself, however, both as to organization to entation are interactively provided to model worksta- 
and method of operation, together with further ob- tion 100 by tracking device 50. The location and ob- 
jects and advantages thereof, may best be under- entation may also be provided manually to model 
stood by reference to the following description taken -. workstation(s) in different embodiments, 
in conjunction with the accompanying drawing in , . Model workstation 100 processes the 3D yoIih 
which: • ; • f 5 . metric data it receives and creates selected render- 
Fig. 1 is a simplified block diagram of a first en> ings of the data. One rendering method determines 
bodimentof a medical display apparatus according to surfaces between differing types of tissue. Connec- 
the present invention. . tMty of similar types of tissue adjacent to one another 
Fig. 2 is a simplified block diagram of a second is then determined. Differentiation, of tissue types 
embodiment of a medical display apparatus accord- 20 based on the nature of the signal in the three- 
ing to the present invention. dimensional image data is known as segmentation. 

When the 3-D volumetric data has been segmented! 

DETAILED DESCRIPTION OF THE INVENTION into internal structures, each interna] structure may 

be treated as a separate solid object by model work- 
in Fig. i, a patient 1 on which a medical proce- 25 station 100; The model workstation has the capability 
dure such as surgery is to be performed, is scanned ... of selectively displaying desired internal structures, 
by a medical imaging apparatus 10 which may be a color coding structures and severing, rotating and 
magnetic resonance (MR) imaging apparatus, a com- translating internal structures in order to manipulate 
puted axial tomography (CAT) apparatus, a positron these images in a desired manner to provide visuali- 
e mission tomography (PET) or similar imaging device . 30 zation to an operator working model workstation 1 00. 
capable i of creating multi- dimensional volumetric : An alternative rendering method generates two- 
data such as 3-dirhensibnaf (3-D) data, from internal dimensional projections of selected features within 
structures of the patient. After imaging, apparatus 10 the three-dimensional data set is described in Euro- 
provides me volumetric data to a model workstation pean Patent Application 0506302 "Projection Meth- 
100. Once the volumetric data has been provided to 35 ods for Producing Two-Dlmensional images from 
model workstation 100y further need for imaging ap- Three-Dime nsional Data". For example, two-dimen- 
paratus IP imaging apparatus is no longer required. \ ■ sional projection angiograms can be extracted from a 
This is important since some medical procedures : three-dimensional phase contrast or time-of-f light 
need not be performed with the patient situated within magnetic resonance angiogram. Several projection 
the confines of an imaging apparatus, which can be 40 algorithms are possible. These include the detection 
constricting in the case of MR imaging. In alternative of the maximum pixel intensity along a selected pro- 
embodiments, imaging apparatus 10 may be interao . jection ray through the three-dimensional data, deter- 
tivefy employed during the medical procedure. Model [■/: mination of the average pixel intensity of a selected 
workstation 100 stores the volumetric data and ere- .. - projection ray arid the determination of the standard 
A : a tes computer genera ted models from the data caps- 45 deviation of all pixels along a selected projection ray; 
ble of .being seated, rotated and otherwise manipuiatT . Model workstation 100 receives input data from 
ed, without the further heed for imaging apparatus a model cut plane input device 40 and a workstation 
V-lP- -:\ ! ' r ; view Input device 60 to select the method of dispIayT 
An operator 350, such as a physician or medical . ; ing Internal structures of patient 1. Model cut plane in- 
' assistant, monitors patient .1;: A semi-transparent so. put device 40 and a workstation view, input device 60 
screen 250 is interposed between patient 1 and op-. . may be a computer pointing device such as a mouse 
erator 350. A tracking device 50 which monitors and or trackball, or any input device which indicates 
tracks operator 350, semi-transparent screen 250; planes in which to cut the images and a viewing angle /, 
and patient 1 determines a relative roll a, pitch 8, and • and scale. Tracking device 50 provides relative orien- 
yaw 4 ; orieritBJbori between operator 350, semi- 55 tation data between operator 350 and patient 1 arid 
transparent screeri 250 ajnd subject 1 . Tracking de- scaling, which allows model workstation 1 00 to syn- 
vice 50 may be a 6-degrees of freedom tracking de- . thesize an interactive computer generated image of 
vice as described "TheF|ockdf Birds" Installation and v internal structures of patient 1 and have ft coincide: 
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with the real-time scene viewed by operator 350. This 
interactive computer generated image may be con- 
verted from a computer monitor signal, such as an 
RGB computer monitor signal, to a video format by 
passing it through a scan converter 192 (shown in 
phantom), depending on the display format of semi- 
transparent screen 250. The computer generated im- 
age is provided to semi-transparent screen 250 which 
is interposed between patient 1 and operator 350. 
Semi-transparent screen 250 provides a desired mix- 
ture of external structures of patient 1 seen by oper- 
ator 350 and the computer generated image from 
model workstation 100. Semi-transparent screen 250 
may receive input signals f rom the operator, for exam- 
ple, through workstation view input 60. This may in- 
volve the degree of transparency of each image, or 
any of various other special effects. One very useful 
special effect is a movable window which has 0% 
transparency (100% opaqueness) superimposed 
upon another image. When the window image is of in- 
ternal structures superimposed upon external struc- 
tures, it creates the illusion of external structures cut 
awaywithih the window, exposing underlying internal 
structures. Other conventional video special effects 
may also be employed. Semi-transparent screen 250 
allows operator 350 to visualize both internal and ex- 
ternal structures simultaneously. The resulting scene 
witnessed by operator 350 is an interactive real-time 
image of patient 1 , even if the patient moves during 
the medical procedure. Since internal structures and 
their relation to exposed external structures are si- 
multaneously displayed, a surgeon using the present 
invention will perceive a very accurate indication of 
where he should cut through external structures to ar- 
rive at a desired internal structure while avoiding vital 

internal structures. 

Figure 2 illustrates an alternative embodiment oi 
the present Invention in which operator 350 receives 
a stereoscopic view of internal and external struc- 
tures of patient 1 . Each of the eyes of operator 350 
differs in its orientation with relation to the patient 
therefore a different view is provided to each eye. 
Tracking device 50 tracks the relative location 
(xt.yi.zd and orientation angle (a,. 4>t» »i) between a 
f irst eye of operator 350 and patient 1. Tracking de- 
vice 50 also tracks a second location (x 2 ,y2,z 2 ) and 
orientation fe.fc.e*) between a second eye of oper- 
ator 350 and patient 1. The second location and on- 
entatlon can be measured independently by tracking 
device 50; or the location and orientation of the first 
eye can be used to compute the location and orienta- 
tion of the second eye. The locations and orientations 
are fed to a first model workstation 100a. and a sec- 
ond model workstation 100b which create a right and 
left computer graphic image at locations (x 1r y$^i). 
fey***), respectively, and orientations (a,, $1. W 
(ai.Mj), respectively, corresponding to the views of 
each eye of operator 350, respectively. Control sig- 



nals from model cut plane 40, workstation view input 
60 and imaging apparatus 1 0 can be sent to both first 
model workstation 100a, which in turn propagates the 
control signals to second model workstation 100b via 
5 communications path 140, or alternatively the con- 
trol signals can be sent directly to both model work- 
stations directly. 

The left and right computer generated image, 
pertaining to a left and right view, respectively, are 
10 converted to video format if required by semi- 
transparent screen 250. This is accomplished by scan 
converters 192a, 192b (shown in phantom) which 
pass the converted computer generated signals to a 
sequencer 198. Sequencer 198 may be a convention- 
's a» video sequencer as described in "Portable Low 
Cost Devices for videotaping, Editing and Displaying 
Field Sequential Stereoscopic Motion Pictures and 
Video" by M. Starks, Stereoscopic Displays and Ap- 
plications Proc. SPIE Vol. 1256, pp. 266-271, 1990. 
2 o Sequencer 198 passes the left computer gener- 

ated image to semi-transparent screen 250. Se- 
quencer 198 then passes the right computer generat- 
ed image to semi-transparent screen 250. Sequencer 

198 alternates many times per second, in synchroni- 
25 zation. between right and left views. 

The image displayed on semi-transparent screen 

\ 250 is time multiplexed to produce an image to the left 
eye and right eye of the operator in an alternating 
fashion. Astereoscopicviewer 252 Is synchronized to 

so a sequencer 198 and operates to block the vision of 
the operator's left or right eye allowing the opposite 
eye to view the image on semi-transparent screen 
250 for an instant and vice-versa. This allows opera- 
tor 350 to see the left image with the left eye while the 

35 right eye sees nothing and the right image with the 
right eye while the left eye sees nothing in rapid suc- 
cession. This creates a stereoscopic illusion, adding 
the dimension of depth perception in viewing thejrn- 
age displayed on semi-transparent screen 250. 

40 Depth perception is very valuable in surgery since it 
adds a dimension that assists in visually localizing 
structures, which is especially important in complex, 

delicate surgery. ■ " . u 'y- 

For both embodiments of the invention the com- 
45 puter image must be registered (coincide) with the ex- 
ternal structures as viewed by operator 350. Inrtiali- 
zationmay be accomplished by manual Input from the 
operator to rotate, translate and scale the computer 
generated image(s) until they coincide with thescene 
50 observed through the semi-transparent screen, or by 
employing tracking device 50 to set Mial parameters. 
Once the 3D model and the visual image of pa- 
- tient tare aligned, tracking device 50 keeps the view 
; angles and-fieid otview consistent This allows real- 
55 time interactive synchronization between the opera- 
'-- tor's view of patient 1 and the computer generated im- 

age(s). : \ •. :/>. 'S' > . KA 

. . Since the image of each Internal structure can be 
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segmented into what resembles a solid object, it may 
be manipulated as a solid object In the case of struc- 
tures of a patient, a surgeon may acquire data from 
the patient by medical imaging, then plan surgery by 
manipulating the models to plan a desired result be- 5 
fore surgery. This is common in complex reconstruc- 
tive surgery. Once the plan Is determined, it may be 
stored and played back during surgery. The images 
of Internal structures are interactively oriented and 
scaled to coincide with the actual patient to 

A user employs as a model cut plane input device, 
• a workstation view input device (40, 60 of Figs. 1 and 
2, respectively) to select planes In which to cut the 
structures in the model, to select a three-di me nsional 
orientation of the model, and to select screen cut 1 s 
planes which define a workstation viewing region. 
. The model workstation can incorporate a clipping cir- 
cuit to determine points within the model cut planes, ■ 
a rotation circuit to rotate points and normal vectors, 
a segmentation processor, and a shading circuit 20 
which determines shading based upon the orientation 
of the normal vector at each point In addition a screen 
clipping circuit can be used to determine points within 
a region defined by the screen cut planes. The model 
workstation also can include a display circuit to create 25 
video signals which, when propagated to a suitable , 
display device, generate images of multiple surfaces 
that are within the desired display region and the 
screen cut planes. 

In radical surgery such as ablative surgery, or 30 
massive trauma cases/ there is little structure which 3 
remains to correctly determine what a normal ana to- - 
my should be. In these cases, an additional model 
workstation may have a model of normal structures 
stored which may be mixed with the other images be- 35 />v, :: ' 
ing displayed to act as a guide in reconstructive sur- -."7 
gery. This may be implemented by additional work- 
stations or model manipulation boards. 

In the' present embodiments of the invention, 
semi-transparent screen 250 is interposed between 40 
operator 350 and patient 1 . Screen 250 can be con- 4. 
structed with a liquid crystal display or it can be com- 
prised of a partially silvered mirror reflecting an image 
. from a video monitor. Semi-transparent display 250 
can be constructed as a relatively large device having is : " 
dimensions approximately equal to that of the region : ; 
of interest of patient 1. or alternatively ft can be of 
small dimension and placed relatively close to the op- 
erator's eyes, perhaps Incorporated into headgear or 
"eyewear. ' • ^'^''A- * 

. . While several presently preferred embodiments 
of the hovel visualization system have been descri- 
bed in detail herein, many rriodifications and yaria- : '':*\ 
tions will now.become apparent to those skilled in the 
art It fe; therefore, to be understood that the append- 55 
ed claims are intended to cover all such modifications } '/ 
.arid variations . : /'.'Vr yC 



Claims 

1 . A real-time medical apparatus for displaying to an 
operator interactive images of internal structures 
of a patient coinciding with a view by the operator 
of the patient comprising: 

a) a medical imaging system for obtaining 
multi-dime nsional imaging data of internal 
structures of said patient; 

b) a workstation for creating images from the 
imaging data of internal structures of said pa- 
tient viewed from a location (x,y,z) and orien- 
tation (ct,4»,u) which coincide with said view of 
the patient by said operator; 

c) a semi-transparent screen adapted for al- 
lowing said operator to view said patient 
through the semi-transparent screen and 
adapted for displaying the images of internal 
structures from the workstation with a desired 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient 

2. The real-time medical apparatus of claim 1 fur- 
ther comprising a tracking device adapted to 

measure the location (x,y,z) and orientation 
(a,$,8) of said operator with respect to said pa- 
tient for repeatedly providing these measure- 
ments to the workstation. 

3. The real-time medical apparatus of claim 1 
wherein the workstation further comprises surgi- 
cal fanning means adapted to create models of 

. ': internal structures of said patient, interactively 
manipulate the models to result in modifications 
to internal structures, store the models and mod- 
ifications and display the models and modifica- 
•: tions viewed from a location and orientation to co- 
inclde with the view of the patient by the operator. 

4. The real-time • medical apparatus of claim 1 
v; wherein the workstation incorporates normal 

anatomical models of- internal st ructures, and is 
adapteo* to display the models viewed from a lo- 
: cation arid orientation to coincide with the view of 
• the patient by the operator, such that said work- 
station can act as a guide; in reconstructive sur- 
■7 V Q€^^;^M\-y-; : {} .-v.. ... ■ 



A real-time medical apparatus for displaying to an 
operator interactive three-dimensional (3D) interr 
rial and external images of a patient comprising: 

a) a medical imaging system for obtaining 
three-dimensional (3D) imaging data of. inter- 

; . nal structures of said patient; 

b) tracking mparts for measuring locations 
' (Xiiyiii) Otiyiz^ and orientation angles 

(<ii4i$d {&2*fc^ <ti each eye of said opera- 
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tor with respect to said patient; 

c) a first workstation for creating a right com- 
puter generated image of internal structures 
of said patient from the 3D imaging data from 
the medical imaging system as viewed from 
the location (x,,y v z,) and orientation angle 
(cti.to.0i) obtained from the tracking means; 

d) a second workstation for creating a left 
computer generated image of internal struc- 
tures of said patient from the 3D imaging data 
from the medical imaging system as viewed 
from a location (x 2 ,y 2 ^2) and orientation angle 
(a 2 ,<k,92) obtained from the tracking means; 

e) a semi-transparent screen adapted to allow 
said operator to view said patient through the 
semi-transparent screen and adapted to dis- 
play the left and right computer generated im- 
ages of internal structures from the worksta- 
tions with a desired degree of transparency in 
rapid succession; 

f) stereoscopic display means synchronized 
with the semi-transparent screen for allowing 
only the operator's right eye to view the semi- 
transparent screen when the right computer 
generated image is being displayed, and for 
allowing only the operator's left eye to view 
the semi-transparent screen when the left 
computer generated image is being displayed 
thereby simulating a 3D image of internal and 
external structures of said patient. 

A method of aiding an operator to perform a med- 
ical procedure upon a patient comprising the 
steps of: 

a) acquiring multi-dimensional medical imag- 
ing data from internal structures of said pa- 
tient; 

b) measuring a location (x,y,z) and orientation 
angle (a.^O) of said patient as viewed by said 
operator; 

c) creating a computer generated image of the 
internal structures from the medical imaging 
data viewed f rom the location (x,y,z) and on- 
entation angle (o4,0); 

d) interposing a semi-transparent screen be- 
tween said operator and said patient allowing 
said operator to view said patient through the 
semi-transparent screen; and 

e) displaying the computer generated image 
on the semi-transparent screen with a desired 
degree of transparency to create the illusion 
of internal structures superimposed upon 
said patient to assist the operator in said med- 
ical procedure. : . 

The method of aiding an operator in a medical 
procedure of claim 6 wherein the step of acqu iring 
medical imaging data Is performed in real-time by 
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a medical imaging system. 

8. The method of aiding an operator in a medical 
procedure of claim 6 further comprising the step 
of creating a computer generated guide image of 
normal anatomical internal structures viewed 
from a location (x.y.z) and orientation angle 
M.B) to be used as a guide to reconstructive 
surgery; and mixing portions of the guide image 
with the computer generated images. 

9. A method of aiding an operator in a medical pro- 
cedure of a patient comprising the steps of: 

a) obtaining three-dimensional (3D) imaging 
data of internal structures of said patient; 

b) interposing a semi-transparent screen be- 
tween said operator and said patient allowing 
said operator to view said patient through the 

semi-transparent screen; 

c) measuring the locations (x^yn^) (x 2 ,y 2 ^2) 

and orientation angles (a* , to ,0i) (a 2 ,to»&2) be- 
tween said patient and each eye of said oper- 
ator, respectively, 

d) creating a left computer generated image 
and a right computer generated image of in- 
ternal structures from the 3D imaging data of 

said patient viewed from distances (Xi,y t ,z,) 
(x 2i y 2fZ2 ) and orientation angles (a t ,to» e i) 
(a2.to.O2) respectively; 

e) displaying the left and right computer gen- 
erated images of internal structures of a de- 
sired degree of transparency in rapid succes- 
sion on the semi-transparent screen; and 

f) blocking the view of the left eye of the op- 
erator when the right computer generated im- 
age is being displayed, and blocking the view 

pf the right eye of the operator when the left 
• computer generated image is being displayed 
thereby simulating a 3D image of internal and 

external structures of said patient 

10. The method of aiding an operator in a medical 
procedure of claim 9 further comprising the step 
of adjusting the distances (x^y^) (x 2 ,y 2 ,Z2) and 
orientation angles (a,,to.0t) (a2.to.O2) of the left 
and right computer generated images to corre- 
spond to the view of the patient by the operator. 
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